Skip to content

Conversation

@olamilekan000
Copy link
Contributor

Problem

In production environments with large datasets, Prometheus metrics collection was timing out due to expensive aggregation queries running directly against large tables. The queries were taking longer than the configured timeout, resulting in failed metric collections and incomplete monitoring data.
This often results into error like

Failed to collect metrics data: failed to query event queue metrics: timeout: context deadline exceeded

Solution

This PR introduces Materialized Views for metrics collection to significantly improve query performance, along with a configurable query timeout.

@olamilekan000 olamilekan000 force-pushed the add-materilized-views-for-prom-aggregatoion-queries branch from 9819daf to 0e488b2 Compare January 6, 2026 08:11
@jirevwe
Copy link
Collaborator

jirevwe commented Jan 6, 2026

We should run (using EXPLAIN ANALYSE) these queries against the US prod db to verify they use an index on the event deliveries table.

@jirevwe jirevwe force-pushed the add-materilized-views-for-prom-aggregatoion-queries branch from 09d0870 to 0e488b2 Compare January 6, 2026 13:14
@olamilekan000 olamilekan000 force-pushed the add-materilized-views-for-prom-aggregatoion-queries branch from 0e488b2 to c4269a8 Compare January 7, 2026 08:55
@olamilekan000 olamilekan000 requested a review from jirevwe January 7, 2026 08:56
@olamilekan000 olamilekan000 force-pushed the add-materilized-views-for-prom-aggregatoion-queries branch 7 times, most recently from c342e4b to 138a721 Compare January 12, 2026 14:10
@olamilekan000 olamilekan000 force-pushed the add-materilized-views-for-prom-aggregatoion-queries branch from 138a721 to 1da3da0 Compare January 13, 2026 02:49
@convoy-engineering convoy-engineering enabled auto-merge (squash) January 13, 2026 10:50
@convoy-engineering convoy-engineering merged commit b36aa3e into main Jan 13, 2026
7 checks passed
@convoy-engineering convoy-engineering deleted the add-materilized-views-for-prom-aggregatoion-queries branch January 13, 2026 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants